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Background / Context: 

Enhancing the reading performanee of at-risk elementary students is one of the thorniest 
problems in Ameriean edueation. In 2009, fourth-grade seores on the National Assessment of 
Edueational Progress were only slightly higher than they were in 1992 (NCES, 2010), and in faet 
have ehanged little sinee 1980. Eurther, there are substantial gaps in reading profieieney 
aceording to soeial elass and ethnicity. Among fourth graders not eligible for free luneh, 45% 
seored at or above proficient, in comparison to only 17% among fourth graders eligible for free 
luneh. The pereent profieient rates were 42% for white fourth graders, but only 16% for African 
Americans, 17% for Hispanics, and 20% for American Indians. 

Success for All (SEA), a whole-sehool turnaround model for elementary sehools that 
foeuses primarily on ensuring that every ehild sueeeeds in learning to read throughout the 
elementary grades (Slavin, Madden, Chambers, & Haxby, 2009)., reeeived one of four Investing 
in Innovation (i3) seale-up grants. Its goal is to help hundreds of struggling sehools by reaehing 
direetly into the heart of praetiee — the interaetions between teachers and students — to improve 
daily lessons and sehool funetioning. 

Purpose / Objective / Research Question / Focus of Study: 

The main eonfirmatory researeh question guiding the design of the impact evaluation is: 
What is the impact of SFA on elementary school students ’ reading achievement, compared to 
students in non-SFA schools? An answer to this question will determine our assessment of 
whether SEA is sueeessful at turning around low-performing sehools. In addition to the main 
eonfirmatory question, this evaluation will address exploratory questions sueh as subgroup 
impaets and impact on non-cognitive outcomes. These analyses are intended to deepen our 
understanding of the overall average impaet of SEA. 

The implementation study has three prineipal goals: to measure the fidelity with which 
the program model is put in plaee (and to determine whether the identity of the SEA eoaeh 
affeets fidelity), to assess the treatment-eontrol eontrast in edueational experienee, and to 
doeument the implementation process and lessons for scale-up and replieation. 

Setting: 

During the 2010-11 sehool year, the SEA team reeruited 37 Title I sehools from five sehool 
distriets loeated in the Southwestern, Mid-Atlantie, and South Central regions of the eountry to 
partieipate in the evaluation. 

Population / Participants / Subjects: 

The 37 schools in the evaluation have been randomly assigned to treatment (n = 19) or control (n 
= 18) conditions. About 3,000 kindergarten students in these sehools were assessed on reading 
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skills in the fall of 2the 2011-12 school year; they will then be followed for three years, with 
assessments taking place each spring. 

Intervention / Program / Practice: 

The main elements of SFA are: 

• School staff buy-in (usually indicated by a faculty vote). 

• Coaches who work with administrators and teachers to implement SFA. 

• The presence at each school of an SFA facilitator who helps all teachers with program 
implementation, ongoing professional development, and school-wide assessments. 

• Extensive professional development for all school staff to help them understand and use 
research-proven approaches to reading instruction, cooperative learning, classroom 
management, motivation, teaching of metacognitive skills, and assessment. 

• A K-6 reading program that uses extensive cooperative learning in pairs and small groups 
to build phonemic awareness, phonics, comprehension, vocabulary, and fluency. 

• Frequent criterion-referenced, instruction-based formative assessments to make sure that 
all students are on track toward success, along with quarterly benchmark assessments to 
track progress in meeting grade-level expectations. 

• One-to-one or small-group tutoring for students who are found to be falling behind grade- 
level expectations. 

• A Solutions Team, which works to prevent or solve problems that go beyond academics. 

• Leadership development that engages the principal and school leadership team in a 
continuous improvement process based on data analysis, goal -setting, and achievement 
monitoring. 

• Implementation self-assessment checklists for teachers and school leaders and 
implementation benchmarks completed by coaches, which provide data for monitoring 
the quality of program implementation and formative outcomes. 

Significance / Novelty of study: 

The study goes beyond previous evaluations of SFA in two key respects. First, it brings to bear a 
wide variety of implementation data that will enable us to draw more informed conclusions about 
how implementation takes place and the extent to which SFA produces changes in classroom 
practice. Second, the study tests the potential of a lower-cost version of SFA in which local 
district coaches rather than coaches employed by SFA assume responsibility for working with 
participating schools. The study will examine whether high-quality implementation is 
maintained with the use of these local personnel, an issue with important implications for policy 
and practice since the lower-cost strategy makes the intervention more readily affordable to 
economically hard-pressed school districts. 

Statistical, Measurement, or Econometric Model: 

Our basic impact estimate will be a two-level HEM model with students nested in schools. 
Students cannot be nested in classrooms, as students are regrouped every quarter and may have 
several reading teachers in a year. Blocking will account for any stratification in the school 
lotteries should districts request this. Covariates in the impact model will include key student 



SREE Spring 2012 Conference Abstract Template 



2 




characteristics, such as percentages of ELL, speeial edueation, and free/reduced priee lunch 
students and baseline student reading achievement test scores. This model will provide an 
intent-to-treat estimate of the impaet of providing aceess to the intervention on students in the 
average sehool in the sample. 

Usefulness / Applicability of Method: 

The methodology, whieh relies on randomization of schools, is justified by the whole-sehool 
nature of the SLA reform. Preliminary data indicate the baseline equivalence of students at T 
and C sehools on measures of early literaey. 

Research Design: 

In the 37 T and C sehools, we will examine two groups of students: 

• Lall 2011 kindergarten students in the randomly assigned schools will comprise the 
primary study sample, so designated beeause they will have been exposed to SLA from 
the beginning of their formal educational experience. They will be followed for three 
years through the end of the 2013-14 school year, when they should have completed 
seeond grade. Sinee the analysis focuses on the sehools in the sample, we will not follow 
students who move away from their original study sehool, but will include “in-movers” 
who join the target grades over time. We will collect annual data on the composition of 
students in both the treatment and control schools to check for any unexpected effects on 
student mobility and, if there are none, we will also be able to examine impaets for a 
“stable sample” of students who remain in the SLA and eontrol schools over time. 

• Students in third, fourth, and fifth grades will constitute an auxiliary study sample. We 
consider this to be an auxiliary sample because students will have been exposed to SLA 
but will not have received it since the beginning of their sehool experience. They would 
be assessed in the spring of the 2012-13 and the spring of the 2013-2014 sehool year, at 
whieh point students in these grades would have had two and three years of exposure to 
the program, respeetively. 

Data Collection and Analysis: 

The Impact Study: The primary student outcome is students’ achievement in reading. Lor the 
primary sample, we plan to use different developmentally-appropriate measures of reading 
achievement. In the fall of 201 1, kindergarten students were individually pretested on the 
Peabody Picture Voeabulary Test (PPVT) and on Woodcock-Johnson III Letter-Word 
Identifieation. Their seores will be used as covariates in the impact analysis. In the spring of 
2012, we will field individually-administered follow-up tests using the Woodcock Letter-Word 
Identifieation and Word Attaek seales. In the spring of 2013 (when students are completing first 
grade) and 2014 (when they are eompleting second grade), we will field individually- 
administered Woodcock Letter-Word Identifieation, Word Attack, and Passage Comprehension 
assessments, and the DIBELs, a reading fluency measure. 

Two group-administered measures of reading aehievement would be taken for the auxiliary 
sample. The first is the Gates-MaeGinitie Reading Test, which provides a nuanced indieator of 
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students’ reading levels and is therefore proposed as the primary measure. Beeause of the poliey 
importanee of state assessments, we will also make arrangements with the study distriets to 
obtain state reading test data for older-grade students and will treat these as a seeondary outeome 
to see if our findings for older students based on the Gates-MaeGinitie are eonfirmed using 
another measure. To deal with the variation in tests aeross states we will plaee the different tests 
on the same metrie by eonverting them to z seores, as suggested by May (2009). Students’ 
third-grade seores on these measures would be used as eovariates in the analysis of their fourth- 
grade and fifth-grade seores. 

The Implementation Study: Data from multiple sourees will inform the implementation study. 
The Sehool Aehievement Snapshot is the key souree of information on implementation fidelity. 
The Snapshot, whieh will be eompleted for eaeh sehool at the eonelusion of the sehool year by 
the sehooTs SFA eoaeh, eontains 40 items tapping the presenee or absenee of sehoolwide 
struetures assoeiated with the program model, along with 20 items deseribing elassroom 
proeesses (a eonstruet that ineludes both instruetional praetiees and student engagement). We 
will further aseertain whether implementation fidelity is affeeted by whether the SFA eoaeh is an 
employee of SFA or an employee of the sehool distriet trained by SFA. 

The Snapshot will also suggest questions to be asked in surveys of prineipals and teaehers at 
treatment and eontrol sehools in order to understand the contrast in treatment that implementing 
SFA entails. The surveys should provide information about the presenee or absenee in the 
eontrol sehools of struetures and proeesses akin to those that SFA sehools and teaehers are 
expeeted to adopt. 

Teaeher logs will eonstitute a further souree of information on differenees in instruetional 
praetiees at the two sets of sehools. We plan to administer a log developed by Brian Rowan and 
his eolleagues at the University of Miehigan that has been shown in prior researeh to 
differentiate effeetively between instruetion in SFA sehools and in sehools that adopted two 
other speeial reading programs (as well as sehools where no speeial reading intervention was in 
plaee). Logs will be eolleeted from first- and seeond-grade reading teaehers in eaeh of the 37 
study sehools in the spring of 2012, 2013, and 2014, with an expeeted sample of approximately 
48 logs per sehool. 

The prineipal and teaeher surveys will foster our understanding of the implementation process, In 
addition, MDRC researehers plan to eonduet site visits to the program sehools, in the eourse of 
whieh they will interview the prineipal and teaehers providing reading instruetion to elieit their 
perspeetives on SFA and its implementation, the support sehool personnel reeeived, ehallenges 
that arose, and responses that were developed to address them. 

Findings / Results: 

The evaluation is ongoing and at this point we do not have any findings 

Conclusions: 

Not applieable, given the absenee, at this juneture, of impaet and implementation findings. 
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